Generalization from Observed to Unobserved Features

The thesis discusses various aspects of learning from a subset of features to another feature subset in the framework of machine learning. We propose feature to feature learning and generalization as a possible model for clustering, as a method to improve feature selection and to incorporate prior knowledge in supervised learning. We also discuss the possible relation of our models to learning in biological neural systems. The most common machine learning model is learning from examples. In this model, the algorithm learns structures from a nite set of observations, called a training set, and uses the learning results to perform some task. In supervised learning each observation is a pair made up of an instance and a label, and the algorithm learns to map between instances to labels. We expect a good learning algorithm to generalize from the nite training set to new unseen instances, and to be able to accurately predict their labels. The instances are typically represented by vectors of measurements, where each measurement is called a feature. For example, in an optical handwritten recognition task, an instance can be an image of the character and the label the name of the character. The features can be the gray level values of each pixel, or functions of these pixels such as strokes. Another prominent learning model is clustering, where each observation contains an instance without a label. Clustering deals with grouping observations based on their similarity. In general, it is not clear how to measure the quality of clustering, since we do not have a well-de ned prediction task as in supervised learning. In some cases, the goal of clustering is to predict some label which is unknown to the clustering algorithm. However, it is unclear how we can optimize the quality of the clustering to an unknown label. Even worse, the clustering quality depends on the speci c choice of the labels. For example, good documents